Skip to content

feat(runner): berth train --exclusive locks the card in the driver - #318

Merged
T-Crypt merged 1 commit into
nextfrom
T-Crypt/train-exclusive
Nov 13, 2026
Merged

T-Crypt merged 1 commit into
nextfrom
T-Crypt/train-exclusive

Conversation

@T-Crypt

@T-Crypt T-Crypt commented Nov 13, 2026

Copy link
Copy Markdown
Owner

What

berth train --exclusive locks the card in the driver too, on Linux with an NVIDIA card.

  • After the grant and before the command starts, the run puts the granted cards in exclusive-process compute mode
    (sudo -n nvidia-smi -i <UUIDs> -c EXCLUSIVE_PROCESS). Only the training's process can then open a CUDA context
    there. When the run ends, it puts them back to DEFAULT.
  • It refuses to start, and the command never runs, when another compute process is on the card, when no sudo rule
    lets nvidia-smi -c run without a password, or when the card has no NVIDIA UUID.
  • --exclusive belongs to berth train alone. berth run --exclusive is refused.

Why

The training lock (#314, #317) covers everything that asks Berth for the card. A program that opens CUDA without
Berth was only reported as a stowaway, and could still push a training out of memory. Only the driver can stop it.

How

  • acquire returns the grant message, so the run reads the granted cards' UUIDs.
  • exclusive in internal/runner/exclusive.go checks the compute apps, sets the mode through the runner's CLI
    hook, and returns the restore func the run defers.

Proof

  • .claude/skills/berth-verify/proof.sh: full proof OK.
  • TestExclusiveSetsAndRestoresTheMode, TestExclusiveRefusesWhenAProcessIsOnTheCard,
    TestExclusiveRefusesWithoutSudo and TestExclusiveNeedsAnNVIDIACard fake nvidia-smi and assert the argv.
  • TestTrainExclusiveRefusesWithoutAnNVIDIACard runs berth train --exclusive against a real berthd whose card has
    no UUID: refused after the grant, command never started. It fails when the runner skips the exclusive step.
  • Not run against the real card: the operator's training holds it tonight. The live check is the first
    berth train --exclusive run after the sudo rule is in place.

berth train --exclusive puts the granted cards in exclusive-process compute
mode after the grant and before the command starts, so a program that opens
CUDA without Berth cannot open them, and puts them back to the default mode
when the run ends. It refuses to start when another compute process is on
the card, when no sudo rule lets nvidia-smi -c run without a password, and
on a card with no NVIDIA UUID. Linux only.
@T-Crypt
T-Crypt merged commit 1bfd344 into next Nov 13, 2026
4 checks passed
@T-Crypt
T-Crypt deleted the T-Crypt/train-exclusive branch November 13, 2026 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant